|
Author
|
Topic: For the Record----Accuracy Statistic
|
stat
Member
|
posted 01-24-2008 05:44 PM
  
I have noticed that many solid veteran examiners are STILL claiming that the polygraph exam is 98% accurate----I won't name names---you all know who you are. So, if an advertised examiner advertises the more modest but scientifically sound statistics 89-93%, they appear as being of a weaker ability to "detect deception" than those that use the old "flawed" or non-robust research projects from the 70's and 80's.It just seems that we all need to be saying the same thing. We are certainly supposed to be doing the same thing (Zone, MGQT, RI), shouldn't we be advertising the same accuracy numbers? ------------------
"This is our hill and these are our beans."----
Leslie Nielsen as Lt. Frank Drebin, Naked Gun 1988
[This message has been edited by stat (edited 01-24-2008).] IP: Logged |
Barry C
Member
|
posted 01-24-2008 06:28 PM
Well, it's interesting. When Don Krapohl did his research on validated techniques, he found the mean (unweighted) accuracy of the Utah and Federal ZCTs to be 90 and 89%, respectively. How then if our best test is 90% accurate, do we have an average accuracy of 98%?Be careful, though, not to go overboard with this. Lou Rovner, in his research, found the CQT to be 95% accurate, and he often cites his own research, which makes sense since he's doing exams based on what he learned in that research - something many of us can't say. What is also interesting is that when Don and then Don and I did our research on Evidentiary Scoring Rules, we found an average accuracy of below 89% on the ZCTs (and people were all over the map). So, even with a technique that averages 89% accuracy, about half the examiners scoring those tests can't "see" the data to make the correct decisions. It's much the same as determining the average of each member of the MLB's batting average and implying people who swing a bat - you and me - have that average - and I probably can hit about the same wheter my eyes are open or not. IP: Logged |
stat
Member
|
posted 01-24-2008 06:54 PM
I am not the least bit ashamed of stating accuracy figures in the 89-90% area to public officials. I always say, "if the weatherman says there is a 90% chance of rain, you'd better bring your umbrella 'cause its going to rain---or at the very least sprinkle."
I have added that "the more experienced and thoughtful the examiner, the better the accuracy rates will likely be, however, we have no way of differentiating examiners from one another per se."I just wish we could ALL accept the newer numbers and change them without fanfare. I am vividly reminded of the Hot Rod faction of society who consistently (like fisherman) fudge their numbers----if taken for their word, every man has a 8 second car. (8 seconds to run a 1/4 mile) It's just rubbish. IP: Logged |
skipwebb
Member
|
posted 01-25-2008 08:50 AM
Hey Stat,
Something you might consider adding in your explanation is the fact that like many other procedures, ultimate accuracy of the polygraph is not wholly within the control or capability of the examiner but rests in part on the active attention, participation and capability of the examinee to understand and follow the process. Additionally, the availability of and access to all of the case facts can affect accuracy of the procedure. For example, the best radiologist will be less accurate reading an x-ray when the person being x-rayed failed to take a deep breath and hold it as instructed during the collection of the information or the radiologist does not have a prior x-ray with which to compare the current findings.
IP: Logged |
blalock
Member
|
posted 01-25-2008 08:53 AM
The "Weatherman" analogy is great, stat. Thanks for sharing it. I'm gonna use that and the radiology example from Inspector Webb around here with the HR people...Ben [This message has been edited by blalock (edited 01-25-2008).] IP: Logged |
Taylor
Member
|
posted 01-25-2008 09:10 AM
I was told (many times over the years) that a specific issue is about 98% accurate. Also, throughout the years, I have found this number accurate on my DI calls. Of course, I can't confirm the NDI calls but I haven't had information come back at a later date on an SO that stated I was wrong. So if this is directed at me, yes, I have 98% on my web site. So what should be listed?
89-90% 89-93% 95% ??????
Three different people with three different answers. Taylor [This message has been edited by Taylor (edited 01-25-2008).] IP: Logged |
Barry C
Member
|
posted 01-25-2008 09:21 AM
quote:
I always say, "if the weatherman says there is a 90% chance of rain, you'd better bring your umbrella 'cause its going to rain---or at the very least sprinkle."
What does that mean though? It means, 90% of the time when these conditions were observed in the past, it rained, so we expect the chance of rain is high today, but it's not certain. Don't forget, 10% of the time when those conditions were observed it didn't rain. (There's no base rate issue with the weather either.) These accuracy numbers aren't "new." We have heard our critics tell us how ridiculous it was to call, for example, court outcomes ground truth. Does it surprise anybody that a person who was investigate, tested, and later charged with a crime would be convicted of that crime? Even without polygraph that would probably be true. In any event the sample isn't really representative of the entire population of people polygraphed, so of course the figures are going to be high. You can still cite those numbers and argue a 98% accuracy rate, but I don't think it's honest, and it certainly isn't all that scientific when you can virtually pick and choose the data you want to include in your sample. My point about different examiners looking at the same data and getting different results was to point out that it's not the techniques fault an examiner can't see the data necessary to render a correct decision. If I were to use a portion of the data and reasoning I mentioned above, I could argue that the Federal ZCT is like flipping a coin when it comes to identifying the truthful (which is what our critics do). If we had only one of the scorers look at the Marin data, then we could have concluded the test is 96% accurate or 78% accurate - all depending on the examiner used to score the data. So which is it? The fact of the matter is that the data was there as many people found it. Instead we average all the scorers and argue the "polygraph" is (or was, in that case) 86% accurate, which is less than the 89% we saw in the research Don looked at in his Validated Techniques article. IP: Logged |
stat
Member
|
posted 01-25-2008 09:48 AM
GREAT discussion.
No Taylor, I didn't even know you had those numbers on your site. I only viewed your site to copy your picture for that photoshop abomination-----be nice, I still have the original (lol).
I believe I wrote that "solid examiners" are using the ole 98% figure, being sure to not convey disdain or direspect. The problem with such a figure is that it is used as a catch all accuracy rate, even though many practioners such as myself ran a vast majority of multi-issue tests.
I had many colleagues in my state tell contract vendors of their 98% accuracy, and felt dishonest agreeing with that number----which put me in the position to inform---and consequently made "solid examiners" look like liars and blowhards. This is unfortunate. If we could agree on a "bumper Sticker Statement" of our accuracy rates, it would be helpful to our credibility and longevity.Being consistant with our advertised accuracy numbers seems to be a reletively painless and simple way to have some solidarity. We are hard pressed to agree on much else, so why not find some universal agreement and be in step. 
IP: Logged |
Barry C
Member
|
posted 01-25-2008 09:55 AM
Just be honest with the figures. (I don't know who cites 98%).If you use that figure, then cite the research and what types of tests that includes. Don't just say polygraph is 98% accurate. IP: Logged |
stat
Member
|
posted 01-25-2008 10:30 AM
Gee Barry (wink), that's not a very catchy sound byte.IP: Logged |
Taylor
Member
|
posted 01-25-2008 02:57 PM
Okay..... I changed it to: The polygraph is extremely accurate. Different testing formats will yield different percentage accuracy rates. The most reliable is a specific issue examination. There, I avoided the entire percentage debate and can talk to an individual personally to avoid the concerns Eric mentioned. Okay E, you converted me.....delete the photoshop picture! lol IP: Logged |
stat
Member
|
posted 01-25-2008 05:13 PM
Taylor, I am ceaselessly impressed with you----your words are always backed up with action. My kind of examiner.IP: Logged | |
Polygraph Place Bulletin Board
